Questionable Raters + Low Agreement + Inadequate Sampling

Several studies have made positive claims regarding the validity, reliability, and utility of the Occupational Information Network (O*NET). In this first of three studies questioning such claims, I focused on the root cause of many problems regarding O*NET: i.e., the practice of rating overly abstract and heterogeneous occupational units (OUs), collecting ratings on OUs that exhibit substantial interrater disagreement, computing aggregate profiles that eliminate true within-OU variance, then viewing correlations involving the aggregates as proof of reliability and validity. I used actual O*NET data and Monte Carlo simulated raters to demonstrate how aggregation bias (James, 1982) can produce the illusory appearance of high correlations at the OU-aggregate level, even when little or no true agreement exists among raters. Because the cross-group convergence studies cited as proof of O*NET's reliability and validity reported correlations that were comparable to (and in some cases, far lower than) results produced from totally random ratings, O*NET's standing with respect to reliability and validity is questionable. I conclude that when evaluating occupational data quality, a necessary (but not sufficient) condition is that interrater agreement levels must exceed results seen in benchmark comparisons using data of known-unacceptable quality (e.g., when random ratings are combined with a few does-not-apply agreements). However, when vague, unverifiable, hypothetical constructs are directly rated using single-item tests (as in O*NET), even perfect interrater agreement is not sufficient to justify inferences of reliability and validity (although the failure to exceed random-rater benchmarks makes a strong statement regarding the lack of quality). Because O*NET’s aggregation bias problems are caused by its overly abstract title taxonomy, the only solution is to start over, and re-populate the database using occupational titles that exhibit far less true withintitle disagreement. Revising the data collection process to rate observable, verifiable job characteristics (and collect data from objective, job-knowledgeable raters) would also help considerably with respect to being able to cite interrater agreement indices in support of an inference of data quality.